Embodiments of the invention are directed to data processing. In particular, embodiments of the invention are directed to data processing streams, where some of the data in the streams are load shed.
While computer databases have become extremely sophisticated, the computing demands placed on database systems have increased at a rapid pace. Database systems are typically configured to separate the process of storing data from accessing, manipulating or using data stored in the database. More specifically, databases use a model where data is first stored, then indexed, and then queried. However, this model cannot meet the performance requirements of some real-time applications. For example, the rate at which a database system can receive and store incoming data can limit how much data can be processed or otherwise evaluated, which, in turn, limits the utility of database applications configured to process large amounts of data in real-time.
To address this issue, stream based computing and stream based database computing is emerging as a developing technology for database systems. And products are available which allow users to create applications that process and query streaming data before it reaches a database file. With this emerging technology, users can specify processing logic to apply to inbound data records while they are “in flight,” with the results available in milliseconds. Constructing an application using this type of processing has opened up a new programming paradigm that will allow for a broad variety of innovative applications, systems and processes to be developed as well as present new challenges for application programmers and database developers.
A data stream may provide a continuous flow of data tuples from a source processing element to a destination processing element. In some cases, there are several processing elements that the data tuples pass through while traveling from the source to the destination.
In some cases, the data tuples are transmitted such that a bottleneck occurs in processing at one of the processing elements.